Prediction using Machine Learned Constraint Satisfaction Programs

Prediction is a well-researched area for Machine Learning applications. In these tasks, the aim is to predict the value for some unseen characteristic based upon the values of other, seen, characteristics for a given example. Machine learning has been extensively applied to these types of tasks by automating the derivation of a predictive function or a set of predictive rules. This predictive function can then be applied to new examples to estimate attribute values. Many techniques have been turned to this purpose. Inductive Logic Programming (ILP) (Muggleton (1999)), for instance, represents the attributes of given examples in first order logic and uses techniques such as inverse resolution to derive a set of first-order logic rules which can be used to logically deduce an attribute value from other attributes. Artificial Neural Networks (Lippmann (1987)) can be trained to predict attribute values. A net is pre-configured with interconnecting perceptron nodes and the weights associated with these nodes are adjusted to improve predictive accuracy over a training set. The learned artefact in this case is a neural net able to predict one attribute value. Decision Trees can predict attribute values by considering, in some stepwise order, the values of other attributes. One method of learning a decision tree is to apply the ID3 algorithm as implemented in the c4.5 decision tree learning program (Quinlan (1993)). The learned artefact is essentially a conjunction of implications which can be applied to a given example to predict the value for a single attribute. Each of these methods can be characterised by the specific ways they represent the task, the artefact they learn and how they learn it. We consider here another type of predictive artefact together with another method of learning. The artefact we learn is a constraint satisfaction (CS) program, typically used for solving CS problems. A CS problem consists of a set of variables {x1, x2, . . ., xn}, a set of domains of values the variables can take and a set of constraints specifying which values the variables can take simultaneously. A solution to a CS problem is an assignment of values to each of the variables from their domains such that no constraints are broken. They find widespread use in science and industry. CS solvers allow users to specify CS problems in a particular syntax as CS programs and then search for solutions to the problem using a configurable search approach. We have adapted a CS program to be used for prediction. By encoding attributes as variables and machine learning appropriate constraints we obtain a predictive CS program. Given a new example for prediction, we add the values of all known attributes as variable value constraints to the predictive CS program. A CS solver can then determine allowable values, i.e. predictions, for unknown attributes. We machine learn constraints by considering combinations of attribute values, or classifications. By comparing how training examples fall into these classifications we can make conjectures about how different classifications relate to one-another and, through this, derive predictive relationships between the relative values of different attributes, which we then encode as constraints. In addition to a CS solver and machine learner, we use a SAT solver to filter conjectures, improving the efficiency of our CS programs. We believe our approach has some benefits over the approaches we discussed above. Firstly, each of the above processes produces an artefact for predicting only one attribute of a given example. This means that should the user wish to predict for another attribute, they need not re-train a new predictor. By contrast, our learned CS programs can predict for any unknown attributes as it includes constraints learned with respect to all considered classifications. In addition, we believe our learned CS program may be more resistant to missing or corrupted data. This additional robustness is valuable as many real-world applications encounter instances of missing or corrupted data.